Skip to content

feat: move GitLab from EKS to EC2 docker-compose#755

Merged
allamand merged 232 commits into
feature/cloudfront-on-agent-platformfrom
feature/cloudfront-on-agent-platform-without-gitlab
Jul 7, 2026
Merged

feat: move GitLab from EKS to EC2 docker-compose#755
allamand merged 232 commits into
feature/cloudfront-on-agent-platformfrom
feature/cloudfront-on-agent-platform-without-gitlab

Conversation

@allamand

Copy link
Copy Markdown
Contributor

Moves GitLab CE out of the EKS hub cluster onto the IDE EC2 instance via docker-compose, exposed through a CDK-managed NLB + CloudFront distribution.

Changes

  • Disable gitlab ArgoCD addon (enabled-addons, hub-config, platform.yaml)
  • Remove GitLab Keycloak SSO client registration (no SSO for EC2 GitLab)
  • Remove gitlab-nlb + gitlab-distribution from Taskfile.cloudfront.yaml (CloudFront now CDK-managed in platform-engineering-on-eks)
  • Add gitlab:init-ec2 task: wait for GitLab CE readiness, create root token, user1, repos via GitLab API (replaces kubectl exec into the k8s pod)
  • Replace k8s Job wait in clone-repos with CloudFront readiness poll
  • GITLAB_DOMAIN_INT now uses EC2 private IP (private/gitlab-ec2-private-ip) for in-cluster ArgoCD git access
  • Remove git_token from seed-secrets (seeded by CDK deploy-time Lambda)

Companion MR

platform-engineering-on-eks: feat/pr-709-kind-crossplane-gitlab-on-ec2feat/pr-709-kind-crossplane

Closes #754

- Disable GitLab ArgoCD addon (enabled-addons, hub-config, platform.yaml registry)
- Remove Keycloak SSO client for GitLab
- Remove gitlab-nlb/gitlab-distribution Taskfile.cloudfront.yaml tasks
- Add gitlab:init-ec2 task: wait for CE readiness, create root token, user1, repos
- Replace k8s Job wait in clone-repos with CF readiness poll
- GITLAB_DOMAIN_INT uses EC2 private IP for in-cluster ArgoCD git access
- Remove git_token from seed-secrets (seeded by CDK at deploy time)

Refs #754
@allamand allamand force-pushed the feature/cloudfront-on-agent-platform-without-gitlab branch from f5ad84b to ea0b1b1 Compare June 26, 2026 09:24
allamand added 23 commits June 26, 2026 11:28
Added: urls, hub:set-overlay-repo, hub:restart-langfuse, hub:wait-for-full-sync,
secrets-manager:seed-secrets, secrets-manager:seed-observability,
hub:create-mgmt-roles, hub:restart-identity-pods, hub:update,
spokes:enable-crossplane/kro, spokes:create-capabilities,
spokes:disable-crossplane/kro/all, spokes:seed-provider-identity

Removed: idc:setup from install task (moved to workshop:Taskfile.yaml)
- Add parallel install phases (phase1/phase2) to kind-kro-ack install task
- Move gitlab:init-ec2 + gitlab:clone-repos to workshop/Taskfile.yaml
- crossplane-system refs in copied tasks left intentional (Crossplane addon on hub EKS)
- Restore argocd login block in ssm-setup-ide-logs.sh
- Restore progressive-app image_name/service name (rollout-demo → progressive-app)
RGDs ekscluster.kro.run and eksclusterwithvpc.kro.run were Inactive because:
1. ACK SecretsManager controller was missing (rg-eks.yaml references secretsmanager.services.k8s.aws/v1alpha1)
2. ESO was not installed on the bootstrap kind cluster (rg-eks.yaml references external-secrets.io/v1)

Fixes:
- Add ACK_SECRETSMANAGER_VERSION=1.3.1 and install it in ack:install
- Add kro:install-eso-bootstrap task (idempotent) before kro:apply-rgds
…dd argoCdCapabilityRoleArn

- Remove spec.argocdCapability block (not in EksclusterWithVpc RGD schema, causes strict decode error)
- Add missing argoCdCapabilityRoleArn field (required by RGD)
- Remove now-unused IDC sed substitution lines
…lity

Port from feature/platform-cluster-kro-ack:
- rg-eks.yaml: add argocdCapability schema object, replace accessEntryArgoCdCapability
  with conditional argocdCapabilityRole + argocdCapability (EKS Capability with IDC) +
  argocdCapabilityAccessEntry (all guarded by includeWhen enabled==true)
- rg-eks-vpc.yaml: add argocdCapability schema object, pass through to nested EksCluster
- hub:claim: restore argocdCapability block with IDC sed substitutions
Deletes eksclusters.kro.run and eksclusterwithvpcs.kro.run CRDs on the
bootstrap kind cluster. Needed when RGD schema changes remove fields
(KRO breaking-change protection blocks the update otherwise).
Run: task kind-kro-ack:kro:reset-crds, then task install.
Hardcoded 10.0.x.0/24 subnets break when vpcCidr is not 10.0.0.0/16.
Add HUB_VPC_PREFIX var (first two octets of HUB_VPC_CIDR) and use it
in hub:claim subnet CIDRs.
…xist

When the EKS ArgoCD Capability is used, ArgoCD CRDs are already present
on the hub cluster. Helm install then fails with 'CRD already exists'.
Add a second status check: skip if applications.argoproj.io CRD exists.
allamand and others added 4 commits July 6, 2026 18:39
… restart credential-less pods (#8)

#7: ArgoCD's fleet-config-repo (repository) and gitlab-user-repo-creds
(repo-creds) secrets were seeded imperatively once and never refreshed, so a
rotated peeks-hub/secrets.git_token left ArgoCD unable to pull GitLab repos
(manual patch per event). New argocd-gitlab-repo-creds chart provisions both as
ExternalSecrets sourcing git_token (refreshInterval 5m, creationPolicy Owner).
Hub-only (environment=control-plane) and served from the GitHub defaults path so
ArgoCD does not need the very GitLab creds it provisions (breaks the circular dep).

#8: KubeVela apps whose service-account (PodIdentityAssociation) and workload
components render in parallel can start pods before the PIA is active; EKS injects
pod-identity creds only at pod creation, so those pods fail CredentialsNotLoaded
until restarted. New podidentity-pod-restart CronJob (gated enable_kubevela)
deletes pods that lack AWS_CONTAINER_CREDENTIALS_FULL_URI but whose SA has a
PodIdentityAssociation (Crossplane upbound or ACK), so they respawn mutated.
Implemented as a standalone self-healing addon rather than editing the KubeVela
ComponentDefinition CUE (lower blast radius, plain-YAML, validatable).
…Kargo steps

From the crossplane KubeVela-path validation:

#5 (healthcheck 404): set healthcheckPath: /collection/FRONT_PAGE on the Rust
app path-based-ingress trait — the app only serves 2xx under /collection/*, so
the ALB target health check must not use the default "/".

#6 (KubeVela rollout not triggered): the CI image-update step only rewrote
image: — KubeVela only re-renders the Rollout when app.oam.dev/publishVersion
changes. CI now also bumps publishVersion to the image tag (java + non-java
branches; no-op for manifests without the annotation).

#7 (Kargo 'Stage defines no promotion steps'): inline the promotion steps into
each Stage's promotionTemplate instead of a PromotionTask reference, which some
Kargo versions fail to resolve at admission. promotiontask.yaml retained for
reference (no longer referenced).

#8 (Kargo dotted key 'key path not found'): address the KubeVela
app.oam.dev/publishVersion annotation with bracket notation
metadata.annotations["app.oam.dev/publishVersion"] instead of dot-escaping.
Kept continueOnError (kro-variant manifests lack the annotation).

NOTE: #7/#8 are Kargo-version-sensitive and were validated only at YAML level;
runtime verification pending on the live Kargo (#8 stays non-blocking via
continueOnError regardless).
…rRole

The Ray RGD creates PV+PVC for S3 model storage (Mountpoint CSI).
Without these resources in the ClusterRole, kro gets 'forbidden' and
the RayService stays in ERROR.
- Use inline steps instead of task-ref (Kargo v1.7.5 webhook rejects task-ref
  with 'Stage defines no promotion steps')
- Remove publishVersion annotation update (yaml-update cannot address keys
  with dots — neither bracket notation nor dot-escaping works)
- Enable auto-promotion for both dev and prod stages (workaround for webhook
  bug blocking manually-created Promotions)
- Mark promotiontask.yaml as reference-only (no longer applied by deploy-kargo.sh)

Validated end-to-end: CI push → Warehouse detects image → dev auto-promotes →
prod auto-promotes → git repo updated with new image tag.
allamand and others added 23 commits July 7, 2026 09:20
… ALB host wildcard

- kro-ack cluster-mgmt trust: add peeks-spoke-*-ack-capability-role (blocker: spoke ACK AccessDenied sts:TagSession); mirror crossplane
- both providers eks-access-management: PassRole for *-microservice-* app roles (blocker: EKS PodIdentityAssociation on rust-microservice-dev-role)
- appmod-service RGD: correct default host wildcard *.us-west-2.elb.amazonaws.com (was elb/us-west-2 labels swapped -> ALB 404)
Replace the exposure_mode==cloudfront toggle in the platform ingress
templates (keycloak, grafana, jupyterhub, argo-workflows, kubeflow,
backstage) with an explicit `insecure` flag:
  insecure=true  -> HTTP:80 listener, no cert, no host rule
                    (consumer terminates TLS upstream — CloudFront, etc.)
  insecure=false -> HTTPS:443 + ssl-redirect + host rule (ACM on the ALB)

The flag is decoupled from CloudFront: any consumer that fronts the ALB
with a TLS-terminating edge sets insecure=true; customers with their own
domain + ACM leave it false (the secure default).

Backward-compatible during the transition: `insecure` falls back to the
legacy exposure_mode=="cloudfront" check when unset, so current installs
keep rendering identically until the provider/config layers are refactored.

Verified with helm template: all six render HTTP:80 (insecure) and
HTTPS:443+host (secure).
…ontract

Additive step toward the generic exposure model:
- `insecure` (bool, default false): true = HTTP-only ingress, consumer
  terminates TLS upstream; false = ALB HTTPS with ACM.
- `certificateArn` (optional): explicit ACM ARN for the HTTPS listener;
  empty = LBC auto-discovers by host.
- `domain` reworded: consumer-provisioned ingress hostname; the platform
  never creates it.

cloudfront.* fields left in place for now — they're removed together with
the provider CloudFront-provisioning code in a later commit to avoid
breaking the current install path mid-refactor.
…kshop/ folder

Workshop is a CONSUMER of the platform, not a cluster provider:
- move cluster-providers/workshop/{Taskfile,README} -> workshop/
- move solutions/ (Module 1 answer-key RGDs, peeks-specific) -> workshop/solutions/
- update the root Taskfile 'workshop:' include to ./workshop

No behavior change: 'task workshop:*' resolves identically (verified with
task --list). This establishes workshop/ as the single home for
workshop-specific logic so the rest of the repo can be the generic solution.
…n from config

NEEDS LIVE-INSTALL VALIDATION.

The generic provider no longer creates a CloudFront distribution or the
internal platform ALB. Instead:
- domain is consumer-provided: ingress_domain_name := config .domain (always),
  in hub:seed and urls. Removed the private/cloudfront-domain reads + AWS
  distribution lookups.
- exposure toggle is driven by the new `insecure` flag: INSECURE := config
  .insecure; EXPOSURE_MODE derived (insecure->cloudfront/HTTP, else domain/
  HTTPS) so the existing cluster-secret annotation chain the addon charts
  consume keeps working. Seed metadata now also carries insecure + certificate_arn.
- removed the cloudfront: include, the cloudfront:setup-exposure/sync-domain
  calls in install, install:phase1-cloudfront (-> phase1-identity, restart
  identity pods only), and the hub:update-ingress-domain task.

common/Taskfile.cloudfront.yaml is left in place for now — kind-kro-ack still
includes it; it is removed together with the kind-kro-ack decoupling next.

Verified only: YAML parses, task lists 40 kind-crossplane tasks, install
summarizes, no dangling cloudfront: refs. NOT install-tested.
…F taskfile

NEEDS LIVE-INSTALL VALIDATION.

Mirror the kind-crossplane change on the kro-ack provider and remove the
now-unused shared task file:
- delete cluster-providers/common/Taskfile.cloudfront.yaml (no longer
  included by either provider).
- kind-kro-ack: drop the cloudfront: include + cloudfront:setup-exposure/
  sync-domain calls; source ingress_domain_name from config .domain; derive
  exposure_mode from the new `insecure` flag; align cert to top-level
  .certificateArn; seed metadata carries insecure + certificate_arn.

Both providers now assume the consumer provisions the domain + TLS edge; the
platform only consumes `domain` + `insecure`.

Verified only: both Taskfiles parse (yq), root 'task --list' resolves all
includes, kind-crossplane=40 / kind-kro-ack=47 tasks. NOT install-tested.
(stale doc mention of Taskfile.cloudfront.yaml remains in kind-crossplane/README.md)
…rom config

NEEDS LIVE-INSTALL VALIDATION. (companion to the shared CF taskfile deletion in 398f5f9)

- drop the cloudfront: include + cloudfront:setup-exposure/sync-domain calls
- source ingress_domain_name from config .domain; derive exposure_mode from
  the new insecure flag; align cert to top-level .certificateArn
- seed metadata carries insecure + certificate_arn

Verified only: yq parses, task lists 47 kind-kro-ack tasks. NOT install-tested.
.local/ (chat/session scratch artifacts) was accidentally committed via a
broad git add. Untrack it from the branch tip and gitignore it. Local files
are preserved.
NEEDS LIVE-INSTALL VALIDATION.

A generic provider must never shell into a named container. Removed the
`docker exec gitlab-gitlab-1 gitlab-rails runner ... ProtectedBranch`
branch-unprotect from spokes:enable-crossplane/enable-kro in both providers
(4 sites). Branch protection on the fleet/gitops repo is the consumer's
responsibility (the workshop configures GitLab when it provisions it).

Parse-verified only (yq + task --list). NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION.

(b) consumer owns spokes (OAP model): remove spoke enablement + fleet-overlay
wiring from the generic platform install in both providers — drop
hub:set-overlay-repo, install:phase2-parallel, and the post-install fleet-config
pull. Platform install provisions the HUB only + exposes the PlatformCluster
abstraction; consumers declare spokes.

Also rename the opaque install phases:
- install:phase1-parallel  -> install:post-sync  (parallel work after the first
  ArgoCD sync: restart identity pods, prefetch, background observability seed)
- install:phase1b-parallel -> install:finalize   (closing step: restart Langfuse
  + print URLs)

spokes:enable-*/disable-*/set-overlay-repo defs are now unreferenced and move
(generic-ized) to workshop/ next. Parse-verified only. NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION.

Remove the now-unreferenced, GitLab/fleet-config-coupled task definitions from
both generic providers (~1400 lines total):
- spokes:enable-crossplane / enable-kro
- spokes:disable-crossplane / disable-kro / disable-all
- hub:set-overlay-repo
- install:phase2-parallel / phase2-spoke-dev / phase2-spoke-prod

Kept spokes:seed-provider-identity (bootstrap Crossplane provider IAM — platform
mechanism, not GitLab-coupled). Also dropped the spokes:disable-all call from the
destroy flow (spoke teardown is now a consumer responsibility, done before
platform destroy).

The generic providers no longer know about GitLab, fleet-config layout, overlay
wiring, or 'tenant: workshop'. Spoke declaration is consumer-owned (OAP model);
generic-ized equivalents move to workshop/ next.

Parse-verified only (yq + task --list; no dangling task calls; only
spokes:seed-provider-identity remains). NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION.

Counterpart to removing spoke enablement from the generic providers. Move the
GitLab/fleet-config-coupled tasks into the workshop consumer (where GitLab
coupling is legitimate — the workshop owns GitLab):
- workshop:set-overlay-repo (was hub:set-overlay-repo)
- workshop:spokes:enable-crossplane / enable-kro / disable-* / disable-all

workshop:install now, after the platform hub install, wires the fleet overlay
and declares spoke-dev (crossplane) + spoke-prod (kro) into the GitLab
fleet-config — the platform's guarded ApplicationSets reconcile them. Added
DOMAIN + K8S_VERSION workshop globals needed by the moved tasks.

Parse-verified only (yq + task --list; workshop:spokes:* + set-overlay-repo
resolve). NOT install-tested.
… model

- config.schema.json: remove the cloudfront.* block (platform no longer
  provisions CloudFront; gitlabDomain is a workshop concern) and generalize
  the adminRoleName description (drop the WSParticipantRole example).
- common/Taskfile.ray.yaml: prestage-models now takes MODEL_REPO_ID +
  MODEL_NAME (default TinyLlama for back-compat) instead of hardcoding
  TinyLlama — keeps Ray generic; the workshop passes the specific model.

Parse-verified only (json + yq + task --list). NOT install-tested.
…al clone)

NEEDS LIVE-INSTALL VALIDATION.

Implements the Q-D consumption model: the workshop treats the platform as an
external artifact and invokes its own `task install` as a black box.
- platform.ref unset -> platform IS this repo (workshop/ is a subfolder); used
  in place at ROOT_DIR, no clone.
- platform.ref set    -> clone platform.repo@ref into .platform/ (pinned
  artifact), copy the workshop config.local.yaml (a superset of the platform
  contract) into it.
- workshop:install now runs 'cd {{.PLATFORM_DIR}} && task install' instead of
  the in-process task-include, so the platform is a true black box.
- added platform:prepare task + PLATFORM_REF/REPO/DIR globals; gitignore .platform/.

Note: ray/spokes still use provider task-includes (fine for the in-place default;
a pinned-clone consumer would get the parent's ray tasks — acceptable, they only
create AWS resources). Parse-verified only. NOT install-tested.
… insecure model

The README documented the removed cloudfront exposure mode + deleted
Taskfile.cloudfront.yaml (pre-created ALB/NLB, gitlab-distribution, EXPOSURE_MODE).
Replace with the current model: consumer provides domain + TLS; `insecure` picks
HTTP vs HTTPS+ACM; the LBC owns the ALB normally; CloudFront is a consumer concern.
GitLab is a consumer concern; the generic platform shouldn't advertise a GitLab
URL. The workshop's setup-env prints GitLab/platform URLs for the IDE.

NOTE: gitlab_domain_name + git_token remain in the cluster-secret seed metadata
intentionally — they're a consumed contract for the addon layer (git_token = the
gitops-repo credential read by argocd-repo-creds/flux/backstage/cicd; gitlab_domain_name
read by backstage/argo-workflows/keycloak GitLab integration). Neutralizing those
names + sourcing git_token from config is a separate addon-level effort (tracked).

Parse-verified only. NOT install-tested.
…+kubevela

The workshop needs two app-deployment spokes, each running Crossplane (app
infra) + KubeVela (OAM app delivery). Provision BOTH spoke-dev and spoke-prod
via Crossplane so each inherits its environment's enabled-addons overlay
(dev/prod both set crossplane: true + kubevela: true) — a single, reliable
addon-enablement path.

Previously spoke-prod was KRO-provisioned, which does NOT set enable_crossplane/
enable_kubevela in its kro-clusters values, so it would not have gotten those
addons. (KRO-provisioning demo dropped; can be restored with explicit
enable_crossplane/enable_kubevela if a KRO module is wanted.)

Parse-verified only. NOT install-tested.
…om repo root

NEEDS LIVE-INSTALL VALIDATION.

The root Taskfile no longer knows about the workshop:
- removed the workshop: include from the root Taskfile (root is generic-only;
  'task --list' at root shows 0 workshop: tasks).

The workshop is now self-contained and run FROM its own folder:
    cd workshop && task install
- introduced REPO (= parent of workshop = platform repo root, via ROOT_DIR/..);
  all platform paths (config.local.yaml, private/, platform/, .platform) hang
  off REPO. Replaced 29 {{.ROOT_DIR}} refs with {{.REPO}}.
- the 3 root-namespace ray refs (:{{.PROVIDER}}:ray:*) become black-box
  subprocess calls: 'cd {{.PLATFORM_DIR}} && task {{.PROVIDER}}:ray:*' — matching
  how install already invokes the platform. The workshop no longer depends on
  being aggregated by the root Taskfile.

Verified: root has no workshop refs; workshop parses + 'task install' resolves
standalone; REPO resolves to the repo root (probe). NOT install-tested.

Remaining root-level workshop item: create-config.sh (workshop config generator,
externally called by CDK) — move needs cross-repo coordination.
… root)

NEEDS CDK/SSM REFERENCE UPDATE (external).

create-config.sh is the workshop config generator (peeks/WSParticipantRole/
gitlabDomain/feature-branch defaults) — workshop-specific, so it leaves the repo
root:
- git mv create-config.sh -> workshop/create-config.sh
- it still targets the repo-root config.local.yaml (read by the workshop + the
  in-place platform): OUTPUT_FILE defaults to $REPO_ROOT/config.local.yaml
  (SCRIPT_DIR/..), and private/ reads use $REPO_ROOT.
- updated root README references to workshop/create-config.sh.

EXTERNAL FOLLOW-UP: the CDK/SSM SetupIDE bootstrap in the companion repo calls
./create-config.sh at the repo root — it must be updated to workshop/create-config.sh.

Syntax-checked (bash -n). NOT run end-to-end.
…om workshop/

Companion to the create-config.sh move: OUTPUT_FILE defaults to
$REPO_ROOT/config.local.yaml (SCRIPT_DIR/..) and private/ reads use $REPO_ROOT,
so the workshop-located script still generates the repo-root config that the
workshop + in-place platform read. Root README references updated to
workshop/create-config.sh. (The prior commit recorded only the bare rename
because the git add aborted on the stale root pathspec.)
The platform now derives exposure_mode from .insecure (not from an empty
domain). CloudFront-mode config left domain:"" but never set insecure, so
providers defaulted to domain/HTTPS+ACM mode -> broken ingress. Emit
insecure:true so the ALB serves plain HTTP behind CloudFront.
…rom-platform

refactor: decouple workshop/CloudFront from the generic platform (WIP)
@allamand allamand marked this pull request as ready for review July 7, 2026 14:37
@allamand allamand merged commit db37e6e into feature/cloudfront-on-agent-platform Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants